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ABSTRACT 


This  report  documents  the  development  of  the  Automated  Quality  Assurance 
Evaluator  Scheduling  Program  (AQSP).  The  program  was  developed  within  the 
requirements  of  Air  Force  Regulation  400-28  and  is  designed  to  provide  QAEs  a 
tool  to  produce  monthly  random  inspection  schedules  required  to  inspect  a 
contractor's  performance  under  an  Air  Force  service  contract.  The  manual 
process,  which  on  average  takes  from  4  hours  to  4  days  each  month  to  produce  a 
schedule,  has  been  automated  and  now  the  same  schedule  can  be  produced  in 
minutes.  The  program  can  assist  all  QAEs  who  use  random  sampling  as  a  method 
of  producing  a  monthly  inspection  schedule,  and  was  developed  for  use  on  a 
Z-100  microcomputer  with  a  Winchester  hard  disk  capability.  A  user's  guide  is 
available  from  the  AFLMC  to  assist  all  users  of  this  program. 
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EXECUTIVE  SUMMARY 


The  Automated  Quality  Assurance  Evaluator  Scheduling  Program  (AQSP) 
is  designed  to  provide  QAEs  an  automated  scheduling  program  which  will 
accurately  and  efficiently  produce  a  randomly  generated  monthly  surveillance 
schedule  required  by  AFR  400-28.  In  a  manual  mode,  Quality  Assurance 
Evaluators  (QAEs)  routinely  spend  anywhere  from  4  hours  to  4  days  every  month 
producing  inspection  schedules.  The  manual  process  is  paperwork  intensive  and 
requires  precise  follow  through  on  numerous  exacting  procedures;  therefore,  it 
is  not  uncommon  for  QAEs  to  produce  inaccurate  and  or  incomplete  schedules. 
Several  Air  Force  Inspector  General  (IG)  reports  documented  numerous  problems 
with  the  QAE  program.  One  recommendation  made  by  the  IG  was  to  develop  a 
computer  generated  random  sample  program.  The  QASP  is  a  response  to  the  IG's 
recommendat ion  and  it  is  a  generic  program  for  QAEs  to  use  on  a  variety  of 
service  contracts. 

Our  initial  efforts  focused  on  developing  a  program  which  would  be  used 
for  the  refuse  collection  service  contract  at  Barksdale  AFB.  Barksdale  tested 
the  prototype  program  and  made  recommendations  for  improvements.  Also, 
coordination  with  HQ  SAC/LGC,  HQ  USAF/RDCL  and  HQ  AFESC/DE  occurred  throughout 
the  development  phase. 

Our  final  product  allows  a  QAE  to  generate  a  monthly  surveillance  schedule 
in  5  to  30  minutes  on  average  versus  the  4  hours  to  4  days  required  under  the 
manual  process.  To  generate  a  schedule,  the  QAE  needs  only  to  answer  some 
very  basic  computer  prompted  questions.  Then  the  QAE  can  request  a  summary 
calendar  output  product  which  shows  how  many  inspections  are  scheduled  for 
each  day  in  the  month,  and  a  monthly  or  daily  checklist  showing  what,  when, 
and  where  to  inspect  a  particular  service  item  on  a  particular  day.  The 
program  uses  a  menu  system  and  contains  detailed  HELP  information  to  assist  if 
needed.  We  estimate  the  average  QAE  who  is  totally  unfamiliar  with  computers 
can  be  fully  proficient  with  1  to  2  days  of  training.  Detailed  user's  and 
programmer's  guides  have  also  been  developed  to  further  assist  all  users  of 
the  program. 

We  recommend  implementation  Air  Force  wide. 
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CHAPTER  1 


PROBLEM 


Background 

AFRs  70-9  and  400-28  prescribe  standard  Air  Force  Quality  Assurance 
Evaluation  (QAE)  program  requirements.  The  AF  Inspector  General  has 
documented  many  problems  with  the  QAE  program  and  recommended  "a 
computer-generated  random  sample  program  should  be  developed..."  as  one  of 
several  solutions.  This  same  recommendation  was  included  in  COPPER  90,  an 
AFLMC  report  which  outlined  initiatives  needed  to  transition  Base  Contracting 
into  the  1990's.  The  Air  Force  expends  a  significant  portion  of  its  annual 
budget  on  a  variety  of  service  contracts  designed  to  support  the  various 
missions  of  bases  worldwide.  As  such,  the  performance  of  QAEs  is  crucial  to 
the  effective  management  of  Air  Force  service  contracts.  QAEs  help  the  Air 
Force  get  what  it  pays  for.  This  project  is  just  one  of  several  identified 
enhancements  necessary  to  improve  the  Air  Force's  QAE  program. 

Problem  Statement 


Monthly  Quality  Assurance  Surveillance  Plans  for  random  sampling  of  Air 
Force  service  contracts  are  difficult  and  time  consuming  to  develop  properly. 
The  exacting  detail  required  by  AFR  400-28  and  extensive  use  of  random  number 
tables  have  caused  many  QAEs  to  have  difficulty  complying  with  requirements  of 
the  regulation.  Therefore,  a  computer  program  which  performs  all  of  the 
tedious  processes  to  produce  an  accurate  monthly  schedule  is  the  recommended 
solution. 


CHAPTER  2 


DEVELOPMENT 


Approach 

Our  original  programming  design  focused  on  developing  a  random  number 
generator  program  which  would  produce  an  inspection  schedule  for  a  refuse 
collection  service  contract.  In  order  to  accomplish  this  objective,  we 
studied  the  requirements  of  AFR  400-28  and  talked  to  a  variety  of  people 
concerning  the  many  manual  QAE  functions  automation  could  eliminate.  We 
concluded  a  QAE  must  perform  six  basic  manual  steps  to  produce  a  monthly 
schedule  accurately  each  month. 

The  oversimplified  steps  for  manually  producing  a  monthly  schedule  are 
depicted  graphically  in  figures  1  through  6  below: 

Using  the  refuse  contract  at  Barksdale  AFB  as  an  example,  a  QAE  must  first 
list  all  of  the  service  locations  required  by  the  contract  and  the  monthly 
frequency  for  performing  each  service. 


Fig  1  -  List  Service  Loc.it  ions 
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Fig  2  -  Determine  Lot  Size 


The  third  step  is  extremely  time  consuming,  because  the  QAE  must  list  all 
4,000  plus  service  locations  on  paper  and  assign  a  sequential  number  to  each 
one . 
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Fig  4  -  Select  Random  Numbers 


Step  5  requires  the  QAE  to  match  the  selected  random  numbers  (In 
Barksdale's  case,  a  total  of  80  numbers  were  required)  against  the  list  of 
4,000  plus  service  locations  identified  in  Step  3. 
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Fig  5  -  Matching  Random  Numbers 

The  final  step  involves  the  actual  preparation  of  the  monthly  schedule. 
The  QAE  must  use  a  calendar  to  pinpoint  exactly  what  lates  the  inspections  for 
the  randomly  selected  service  locations  fall  on.  The  QAE  then  prepares  a 
tally  checklist  for  use  during  the  inspections.  These  six  steps  complete  the 
tedious  and  exacting  routing  required  by  QAEs  to  generate  a  schedule. 
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Fig  6  -  Preparing  a  Monthly  Schedule. 

This  process  takes  the  average  QAE  anywhere  from  4  hours  to  4  days  every 
month  to  produce  a  schedule.  Our  programming  design  effort  was  geared  to 
maximize  automation  of  the  process  and  thus  simplify  it.  This  in  turn  will 
produce  more  accurate/complete  schedules  and  reduce  the  time  required  by  the 
QAE  to  produce  the  schedule. 

From  the  outset  of  program  development,  we  focused  on  making  the  program 
easy  to  use  and  kept  this  at  the  forefront  of  testing  and  demonstration.  We 
demonstrated  the  prototype  program  at  Barksdale  AFB,  LA  on  their  refuse 
collection  service  contract.  As  a  result,  two  enhancements  were  identified  to 
further  improve  the  program.  They  include  developing:  (1)  a  contractor 

performance  management  system,  complete  with  notices  when  Acceptable  Quality 
Levels  (AQLs)  are  exceeded;  and  (2)  a  QAE  management  data  base  to  track  QAE 
appointments  and  training  requirements.  We  also  demonstrated  the  prototype 
program  at  Tyndall  AFB,  FL  for  the  benefit  of  the  Air  Force  Engineering  and 
Service  Center  (AFESC)  representatives,  since  the  engineering  and  services 
functions  will  be  the  biggest  users  of  the  program. 

After  our  initial  reprogramming  effort,  we  revisited  Barksdale  and 
Tyndall  AFBs  to  demonstrate  the  revised  program.  To  reinforce  the  emphasis  on 
user  friendliness,  we  also  used  a  QAE  from  Maxwell  AFB,  AL  to  test  the 
program. 

After  providing  Barksdale  an  enhanced  version  of  the  program  to  use  on 
their  refuse  collection  contract,  the  QAE  produced  two  consecutive 
problem-free  monthly  schedules.  We  then  coordinated  the  program  with  HQ 
USAF/RDCL,  HQ  SAC/LGC  and  HQ  AFESC/DE,  and  concluded  the  program  met  the 
criteria  established  in  AFR  400-28.  However,  several  additional  tests  were 
conducted  to  reverify  the  ease  of  using  the  program. 
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Additional  tests  on  the  user  s  guide  were  conducted  at  Maxwell,  Tyndall, 
MacDill  and  Andrews  AFBs.  We  concluded  an  effective  user's  guide  should 
include  a  tutorial  section  in  addition  to  specific  reference  information.  The 
QAEs  who  tested  the  program  and  user's  guide  proved  our  products  were 
effective.  As  a  result,  final  software  modifications  to  the  program,  the 
programmer's  guide,  and  user's  guide  were  completed. 

Results 

The  Automated  Quality  Assurance  Evaluator  Scheduling  Program  provides  a 
QAE  the  ability  to  accurately  and  efficiently  produce  a  random  sample  monthly 
inspection  schedule.  Instead  of  spending  many  hours  or  days  to  produce  a 
schedule  manually  each  month,  the  program  can  produce  the  schedule  in  5  to  30 
minutes  on  average.  This  is  a  significant  reduction  in  schedule  preparation 
time;  therefore,  QAEs  will  have  more  time  to  inspect  a  contractor's 
performance . 

Using  the  program  requires  the  following  three  basic  steps.  In  Step  1,  a 
master  file  is  created.  The  master  file  inputs  consist  of:  (1)  the  name  of 

the  service  and  any  additional  services  required  for  the  primary  service;  (2) 
the  AQL  for  each  service;  (3)  the  specific  item  location  and  name  of  item;  (4) 
the  service  days  scheduled  during  the  week  and  the  number  of  times  an  item  is 
serviced  per  day.  In  Barksdale's  case,  a  total  400  plus  contract  line  items 
were  input  per  the  above  referenced  requirements.  Once  all  inputs  are  loaded, 
the  program  will  then  automatically  calculate  the  lot  size,  the  sample  sizes 
(normal,  medium,  small,  reduced)  for  the  lot  size,  and  the  monthly  frequency 
of  each  specific  service  item.  Figure  7  depicts  a  sample  of  a  master  file. 


01/28/86  MASTER  FILE  LISTING  FOR:  INDUSTRIAL  REFUSE  COLLECT 

FOR  MONTH  OF:  Februar- 
START  DAY  OF  MONTH:  Saturday 


Service  AQL 

REFUSE  W/OUT  GARBAGE  2.5 

CONTAINER  SERVICABILITY  2.5 

EMPTY  CONTAINERS  6.5 

POSITION  OF  CONTAINERS  6.5 

SANITATION  6.5 

SPILLAGE  6.5 


Location  Item  Designation 


Un-Acc 

Lot  Size 

Normal 

Medium 

Small 

00 

000760 

0080 

032 

020 

00 

000760 

0080 

032 

020 

00 

000760 

0080 

032 

020 

00 

000760 

0080 

032 

020 

00 

000760 

0080 

032 

020 

00 

000760 

0080 

032 

020 

Freq  Not-Weekly  Per-Day  Alt  5-or- 


001-8122 

TRANSMITTER  SITE 

004 

0 

1 

0 

0 

Acc  Freq: 

000000  Service 

Dates :  03 

10 

17 

24 

002-8930 

BX  CONVENIENCE  STORE  008 

0 

1 

0 

0 

Acc  Freq: 

000004  Service 

Dates :  03 

06 

10 

13 

17 

20 

24 

27 

003-7664 

CULLEN  PARK 

004 

0 

1 

0 

0 

Acc  Freq: 

000012  Service 

Dates:  03 

10 

17 

24 

004-7662 

FORESTRY  OFFICE 

004 

0 

1 

0 

0 

Acc  Freq: 

000016  Service 

Dates :  03 

10 

17 

24 

005-0001 

CLEAR  LAKE  PARK 

004 

0 

1 

0 

0 

Acc  Freq: 

000020  Service 

Dates:  03 

10 

17 

24 

006-7632 

NAVY  COMPOUND 

004 

0 

1 

0 

0 

Acc  Freq: 

000024  Service 

Dates:  06 

13 

20 

27 

007-7835 

RECEIVER  SITE 

004 

0 

1 

0 

0 

Acc  Freq: 

000028  Service 

Dates:  03 

10 

17 

24 

008-7588 

CONVENTIONAL  WSA 

004 

0 

1 

0 

0 

Acc  Freq: 

000032  Service 

Dates:  06 

13 

20 

27 

009-7056 

K-9  KENNELS 

004 

0 

1 

0 

0 

Acc  Freq: 

000036  Service 

Dates:  06 

13 

20 

27 

010-6394 

FAA 

004 

0 

1 

0 

0 

Acc  Freq: 

000040  Service 

Dates:  06 

13 

20 

27 

011-6564 

RIFLE  RANGE 

004 

0 

1 

0 

0 

Acc  Freq: 

000044  Service 

Dates :  06 

13 

20 

27 

012-7428 

WSA  (RESTRICTED) 

004 

0 

1 

0 

0 

Acc  Freq: 

000048  Service 

Dates:  06 

13 

20 

27 

013-731 8 

WSA  (RESTRICTED) 

004 

0 

1 

0 

0 

Acc  Freq: 

000052  Service 

Dates:  06 

13 

20 

27 

014-7322 

WSA  (RESTRICTED) 

004 

0 

1 

0 

0 

Acc  Freq: 

000056  Service 

Dates:  06 

13 

20 

27 

015-7201 

SUPPLY 

008 

0 

1 

0 

0 

Acc  Freq: 

000060  Service 

Dates:  03 

06 

10 

13 

17 

20 

24 

27 

Reduced 

032 

032 

032 

032 

032 

032 

SMTWTFS 

0100000 

0100100 

0100000 

0100000 

0100000 

0000100 

0100000 

0000100 

0000100 

0000100 

0000100 

0000100 

0000100 

0000100 

0100100 


Figure  7  -  Master  File  Output  Product 


Step  2  requires  the  QAE  to  update  the  master  file  for  each  month  a 
schedule  is  desired  by  answering  computer-prompted  questions  such  as:  (1) 

What  month  is  the  schedule  being  prepared  for?;  (2)  What  day  is  the  first  day 
of  the  month?;  (3)  Are  there  any  days  to  be  excluded  from  consideration  for 
random  selection,  etc.?  Also,  any  maintenance  work  required  to  change,  add  or 
delete  any  information  pertaining  to  a  particular  master  file  input  is 

accomplished  at  this  time  in  order  to  make  the  final  step  valid. 

Step  3  requires  the  QAE  to  decide  via  menu  driven  screen  displays  what 
output  products  to  generate.  The  two  main  outputs  available  are  the  monthly 
calendar  which  summarizes  the  total  number  of  inspections  scheduled  for  each 
day  during  the  month  and  the  monthly  schedule/checklist.  The  monthly  summary 
calendar  depicted  in  Figure  8,  provides  a  QAE  with  a  quick  snapshot  of  the 
inspection  workload  for  the  following  month.  The  calendar  indicates  the  total 
number  of  inspections  scheduled  for  each  day  during  the  month. 
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SUMMARY  OF  QAE  MONTHLY  SCHEDULE  FOR:  INDUSTRIAL  REFUSE  COLLECT 
As  of:  1/28/86 

««<  FEBRUARY  1986  »»> 


SUN 

MON 

TUE 

WED 

THU 

FRI 

SAT 

01 

02 

03 

04 

05 

06 

07 

08 

6 

1 

2 

4 

3 

2 

09 

10 

11 

12 

13 

14 

15 

1 

1 

3 

3 

16 

17 

18 

19 

20 

21 

22 

2 

1 

1 

2 

2 

23 

24 

25 

26 

27 

28 

4 

3 

3 

3 

2 

NOTE:  Number  in  lower  right  corner  indicates  the  total  number  of  inspections 

to  be  performed  on  that  day.  "EXCL"  indicates  excluded  date(s). 

************************************************ 

*  FOR  OFFICIAL  USE  ONLY  * 

*  NOT  TO  BE  RELEASED  TO  CONTRACTOR  PERSONNEL  * 

************************************************ 

Figure  8  -  Monthly  Summary  Calendar 


10 


^  >  ■> 


The  monthly  schedule/checklist  is  a  comprehensive  document  which  provides 
the  QAE  a  day  by  day  breakdown  of  specific  services  randomly  selected  by  the 
computer  for  inspection.  The  schedule  indicates  the  actual  random  number 
selected  by  the  computer  and  provides  a  space  (i.e.  the  tally  checklist)  for 
the  QAE  to  annotate  after  inspection  remarks.  Figure  9  depicts  a  sample  of  a 
monthly  schedule. 


QAE  MONTHLY  SCHEDULE  FOR:  INDUSTRIAL  REFUSE  COLLECT 
For  the  month  of:  FEBRUARY 
As  of:  1/28/86 

<<<<<<<  Friday  the  14th  »»»> 


//////////////////////////////////////////////////////////////////////////// 


Service 

REFUSE  W/OUT  GARBAGE 


Sample  Requested 
32 


Location  Item  Designation  Occ  Acc/UnAcc  Ranrabr 

1.  038-6415  HANGAR  (5)  /  190 


Sample  Used  AQL% 
32  2.5 

tanrabr  Remarks 


***  CONTAINER  SERVICABILITY 


***  EMPTY  CONTAINERS 


***  POSITION  OF  CONTAINERS 


***  SANITATION 


***  SPILLAGE 


2.  052-5778  SHOP  ENG  I&R 


***  CONTAINER  SERVICABILITY 


***  EMPTY  CONTAINERS 


***  POSITION  OF  CONTAINERS 


***  SANITATION 


***  SPILLAGE 


3.  113-4314  DORM  PARKING  LOT 


***  CONTAINER  SERVICABILITY 


***  EMPTY  CONTAINERS 


***  POSITION  OF  CONTAINERS 


***  SANITATION 


***  SPILLAGE 


//////////////////////////////////////////////////////////////////////////// 


COMMENTS: 


Figure  9  -  Monthly  Schedule 


Additional  features  of  the  program  include  the  ability  to  generate  a  set 
of  random  numbers  only;  an  AQL  monthly  summary  listing  which  depicts  the 
status  of  a  service  items  accept/reject  level;  and  a  10%  overdraw  feature  to 
complement  the  regular  monthly  schedule  in  case  the  QAE  misses  a  required 
inspection  date. 

Hardware /Software  Requirements 

The  following  equipment  is  required: 

-  Zenith  Z-100/Z-248  microcomputer  with  Winchester  hard  disk  drive. 

-  MSDOS  Version  2.1  or  higher  operating  system. 

-  COBOL  Software  Version  1.12  or  higher.  (Microsoft  COBRUN.EXE  version 
1.12  is  included) 

-  Printer  (any  Z-100  compatible  printer). 

The  AFLMC  will  provide  initial  program  software  and  users/programmers 
guides  to  all  MAJCOM  LG#s  and  DEs. 


-  For  information  on  ordering  additional  copies,  call  AV  446-4085. 


CHAPTER  3 


CONCLUSIONS  AND  RECOMMENDATIONS 


Cone lus ions 


1.  The  Automated  Quality  Assurance  Evaluator  Scheduling  Program  (AQSP) 
provides  QAEs  the  ability  to  accurately  and  efficiently  produce  monthly  random 
inspection  schedules  with  enough  input  flexibility  to  allow  a  QAE  to  perform 
duties  without  yielding  to  the  temptation  to  "pencil-whip"  inspection  schedule 
preparation  responsibilities.  The  program  further  eliminates  the  QAE's 
requirement  for  detailed  written  documentation  as  to  how  the  schedule  was 
produced  which  frequently  involves  burdensome  paperwork.  The  program  does  not 
guarantee  a  contractor's  actual  performance  will  be  measured  more  accurately; 
however,  it  will  provide  an  objective  schedule  which  meets  AFR  400-28  criteria 
and  it  does  solve  the  problem  identified  by  the  Air  Force  IG. 

2.  Because  of  the  proven  benefit  of  this  program,  the  program  should  be 
distributed  Air  Force  wide  and  become  a  standard  Air  Force  system.  Also,  to 
further  upgrade  the  QAE  management  information  system  concept,  an  effort  to 
develop  a  contractor  performance  management  system  and  a  QAE  management  data 
base  should  also  be  considered  for  future  development. 

Recommendat ions 


1.  Approve  use  of  the  Automated  Quality  Assurance  Evaluator  Scheduling 
Program  for  use  by  QAEs  Air  Force  wide.  (OPR:  HQ  USAF/RDC) 

2.  Assist  Air  Force  MAJCOM  functional  representatives  with  training  and 
distribution  of  the  software  and  users  guide.  (OPR:  AFLMC/CC) 

3.  Transition  the  software  to  the  Data  Systems  Design  Office  for  maintenance 

as  a  standard  Air  Force  data  system.  (OPR:  USAF/RDC,  OCR:  DSDO/CC, 

AFLMC/CC) . 
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